Practical Recommendations for Replay-Based Continual Learning Methods

Continual Learning requires the model to learn from a stream of dynamic, non-stationary data without forgetting previous knowledge. Several approaches have been developed in literature tackle challenge. Among them, Replay empirically proved be most effective ones [16]. operates by saving some samples memory which are then used rehearse knowledge during training subsequent tasks. However, an extensive comparison and deeper understanding different replay implementation subtleties is still missing literature. The aim this work compare analyze existing replay-based strategies provide practical recommendations on developing efficient, generally applicable strategies. In particular, we investigate role size value, weighting policies discuss about impact augmentation, allows reaching better performance with lower sizes.